Binaural Audio-Visual Localization

Localizing sound sources in a visual scene has many important applications and quite few traditional or learning-based methods have been proposed for this task. Humans the ability to roughly localize within beyond range of vision using their binaural system. However most existing use monaural audio, instead as modality help localization. In addition, prior works usually form object-level bounding boxes images videos evaluate localization accuracy by examining overlap between ground-truth predicted boxes. This is too rough since real source often only part an object. paper, we propose deep learning method pixel-level leveraging both recordings corresponding videos. Specifically, design novel Binaural Audio-Visual Network (BAVNet), which concurrently extracts integrates features from We also point-annotation strategy construct ground truth network training performance evaluation. Experimental results on Fair-Play YT-Music datasets demonstrate effectiveness show that audio can greatly improve localizing sources, especially when quality information limited.